Touch gesture detection on a surface with movable artifacts

ABSTRACT

Systems and methods for touch gesture detection between two objects wherein a potential touch gesture that was detected by visual analysis of captured images of the two objects is confirmed by a correlated movement of one or more movable surface artifacts on at least one of the objects. A non-confirmed potential touch gesture may be determined to be a hovering gesture. Sound sensors that independently detect and locate finger touches, as well as motion sensors that detect vibration corresponding to actual touch tap events, may be combined with the captured images to improve the detection accuracy. For wearable smart devices like smartwatches, the skin area over the back hand with its movable surface artifacts may serve as a virtual touchpad, accepting touch gestures by the user&#39;s fingers.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Ser. No. 15/859,457 filedDec. 30, 2017, which claims the benefit of provisional patentapplication Ser. No. 62/441,277, filed Dec. 31, 2016 by the presentinventor. All of these applications are incorporated by reference hereinin entirety.

STATEMENTS REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not applicable.

REFERENCE TO A MICROFICHE APPENDIX

Not applicable.

TECHNICAL FIELD

The disclosed embodiments relate generally to a user interface between aman and a machine, and more particularly, to a virtual touchpadsinterface for a wearable and portable devices.

BACKGROUND

Many smart wearable devices have been introduced in the past decade.They provide extensive information by means of visual display or sound,employ smart sensors, use sophisticated processing and communicationdevices, and they are well integrated with the user's other smartdevices.

The smartwatch has become the most popular smart wearable device inrecent years. It allows the user to download, read and communicateinformation, conduct standard phone operations, and monitor their body'sactivities and health using sophisticated sensors. The integration ofthe smartwatch with other popular portable devices (in particular, thesmart phone) brings forth many more sophisticated applications and uses.

While smartwatches have widely adopted the touch screen technology thatmade the mobile phone a great success, the inherently tiny screen foundon a typical smartwatch is frequently too small for selective touchcommands. The finger that touches the screen is always in the way,obstructing the display. Touch screen gestures often cause unintendedactivation of links that may be present on the screen during scrolling,and result in fingerprints and dirt being left on the display. It mayalso be difficult to see which input to make on the small screen,particularly for those with diminished eyesight.

The tiny screen found on a smartwatch is simply way too small to performany common multi-touch commands like pinching, rotation, and othergestures. Placing two fingers on the screen will block most of thedisplay area. Meanwhile, multi-touch gestures remain crucial for theuser interface in many common applications. For example, the user maywant to browse a map or a picture on the small screen of the smartwatch,or change the image scale. Typing data on the small screen is also closeto impossible on wearable devices.

In the prior art, various devices with limited screen capabilities areroutinely interfaced with external tactile input devices like the commonmouse or touchpads. However, connecting such conventional tactile inputdevices is not possible for wearable devices since the device is worn onmoving and often space limited surfaces. User interface solutions likevirtual keyboard projection systems were deployed in recent years forsmartphones, tablets and other mobile devices. However, this solution isnot suitable for wearable devices because the body surface on which thevirtual keyboard will be projected is neither flat nor even, and theskin has a texture and color that may not display projection imagesclearly. In addition, most wearable devices are low profile (i.e. theyhave low height), which makes proper projection of virtual keyboardsclose to impossible.

SUMMARY

The present invention discloses systems and methods to improve the userinterface with wearable devices like a smartwatch. It assigns one ormore virtual passive touchpads on the surface adjacent to the device,where the user can tap, drag, draw characters or perform multi-fingergesture commands which are captured by the device. Thus, generic tactiletouchpad inputs to the device are simulated by the assigned virtualpassive touchpad without the need of placing an actual tactile touchpaddevice on that area.

For a smartwatch, the back hand area near the wrist on which thesmartwatch is worn, is an advantageous choice for a virtual touchpad.However, the back hand area is not perfectly flat. In fact, it isnaturally curved along various directions and has small surfacedistortions (artifacts) due to the blood vessels, wrinkles and skinblemishes. Therefore, touch detection with infrared sensors that arecommon in virtual laser keyboards may not work effectively.

In some embodiments, the touch locations on the assigned virtualtouchpad area are captured by one or more side looking cameras. Thecameras detect one or more finger touch points relative to the capturedimage of the back hand surface. They can also detect complex gesturesincluding hovering, touching (or tapping), and dragging. The cameras maydetect small movements in the surface artifacts to confirm that theuser's finger is actually touching the back of the hand, and not justhovering above it.

In some embodiments, the side looking cameras may be assisted withsound/finger tap sensors and movement sensors. The data from thesound/finger tap sensors is compared with the gesture analysis of thecameras to better pinpoint the position of the user's fingers over thevirtual touchpad. Similarly, the data spikes from the movement sensorwhen the user actually touches the virtual touchpad may be used tocorrelate with the camera's data.

In some embodiments of a wearable device like a smartwatch, only onevirtual touchpad is assigned on the back of the hand since this area isfrequently exposed and available for a user tactile gesture interface.However, more than one virtual touchpads may be assigned. In the case ofthe smartwatch, a second virtual touchpad may be assigned over theuser's arm near the smartwatch. When multiple virtual touchpads areassigned, they may work in an interchangeable operation mode or in acomplementary operation mode. During interchangeable operation mode eachvirtual touchpad can detect the full set of recognized gestures. In thecomplementary operation mode, some of the multiple virtual touchpads maybe restricted to recognize only a subset of the recognized gestures, andsome gestures may have sub-gestures that need to be entered onspecifically assigned virtual touchpads at a predefined order.

In accordance with some embodiments, the touch location over the backhand surface may be actively captured via an analysis of radio signalpropagation delays along the user's body.

The teaching of side viewing cameras of the present invention is alsoapplicable in other embodiments of a general purpose low profile devicewith a small screen that is placed on arbitrary surfaces. Selected areasof the surface surrounding the device may be used as assigned virtualtouchpads.

These and other advantages, objects and features of the presentinvention will become apparent from the accompanying drawings anddescriptions, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the aforementioned embodiments of theinvention as well as additional embodiments thereof, reference should bemade to the Detailed Description of the invention, in conjunction withthe following drawings. It should be understood that these drawingsdepict only exemplary embodiments of the present disclosure and are nottherefore to be considered to be limiting its scope. In the drawings,like reference numerals designate corresponding elements, and closelyrelated figures have the same number but different alphabetic suffixes.

FIG. 1 illustrates a smartwatch demonstrating cursor control by avirtual back hand touchpad in accordance with some embodiments.

FIG. 2 shows how the virtual back hand touchpad may be used to select anaction icon shown on the display.

FIG. 3 shows how the virtual back hand touchpad may be used to entercharacter by drawing the shape of the character.

FIG. 4 illustrates the use of a common multi-touch gesture to increasedisplay magnification.

FIG. 5 is a block diagram illustrating a wearable electronic device thatmay be controlled by a virtual back hand touchpad.

FIG. 6 illustrates cameras for detecting finger touch position on thevirtual back hand touchpad in some embodiments.

FIG. 7A depicts the position of one of the cameras of FIG. 6.

FIG. 7B illustrates a visibility cone of the camera of FIG. 7A inrelation to the back hand area.

FIG. 7C shows how the back hand area may rotate relative to the camerawhen the hand is moved.

FIGS. 8A-8C demonstrates changes in the captured images of the back handtouchpad surface during a touch drag gesture. FIG. 8A captures thefinger hovering above the touchpad, FIG. 8B captures the initial touchof the drag operation, and FIG. 8C captures the finger motion during thedrag gesture.

FIG. 9 is the flow diagram of the analysis of the captured images ofFIGS. 8A-8C.

FIG. 10A shows a use of an array of sound sensors placed at the bottomof the smartwatch that may detect finger tap positions on the virtualback hand touchpad.

FIG. 10B illustrates the position of the sound sensors of FIG. 10A.

FIG. 11 shows a process flow diagram that performs an initialcalibration of the virtual touchpad.

FIG. 12 shows a process flow diagram that combines data from soundsensors, motion sensors and camera data to obtain more accuratedetermination of finger positions on the virtual touchpad.

FIG. 13 illustrates a radio sensor for the detection of finger tappositions on the virtual back hand touchpad.

FIG. 14 details the propagation paths of signals of the radio sensorthat are used to determine the finger touch position.

FIG. 15 shows cameras on both sides of the device allowing command entryfrom both a left virtual touchpad and a right virtual touchpad.

FIG. 16 represents a general case relating to a low-profile device witha display placed on an arbitrary surface where multiple virtualtouchpads are defined.

FIG. 17 is a flow diagram of multiple virtual touchpads operating in theinterchangeable mode, where the user may perform the same gesture onevery virtual touchpad.

FIG. 18 is a flow diagram of multiple virtual touchpads operating in thecomplementary mode, where complementary mode gestures require the userto tap a sequence of virtual touchpads.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 illustrates a user wearing a smartwatch 40 on the user's lefthand 100 having a display (also referred to as “screen view” or “screendisplay”) 42 on which a cursor or a selection marker 82 is displayed.The smartwatch is used only as one example of common wearabledevice—persons skilled in the art should be able to readily adapt theteaching in this disclosure to other wearable devices, and following thediscussion below in conjunction with FIGS. 16-18, to any other portablelow-profile devices that are placed on arbitrary surfaces.

A portion of the back hand 100 surface 80 is assigned as a virtualtouchpad on which touch gestures are detected. Unlike the well-knowntactile touchpad that already has standard interface which isextensively used in the art to control connected devices, the virtualtouchpad must be monitored externally to detect user's touch gestures.The detected gestures must simulate a corresponding tactile touchpadinput that can be readily used by to the smartwatch. The virtualtouchpad 80 is mapped on the back side of hand 100 in proximity to thesmartwatch 40. The virtual touchpad allows the user to interface withthe smartwatch 40 using one or more fingers of the right hand 102. Thesystem detects the finger(s) 116 movement over the virtual touchpad, andsimulates a corresponding standard generic touch touchpad or mouseactivity as an input to the device. When simulating a standard tactiletouchpad input, the virtual touchpad 80 may be directly mapped so that atouch at point 84 on the virtual touchpad 80 causes the cursor 82 to bein the corresponding position. The smartwatch may have one or morebuttons 24 on both sides allowing the user to control some functions inthe traditional way for older digital watches. For clarity purposes,buttons 24 will not be shown in the remainder of the figures of thisdisclosure.

It is well known that the hand surface is not a smooth flat surface.Further, the skin exhibits slight movements and shape changes (e.g.small depressions on the skin surface) whenever the user touches it andperforms gestures with one or more fingers. It has visible blood vessels108, 109, 110 and various stains, wrinkles and blemishes 111, 112. Thevarious bones (not shown in the drawings for clarity) also cause variousridges and depressions on the back hand surface. Other individualelements visible on the back hand may include visible hair, birth marksand other textures. All these elements create a challenge for trackingthe finger touch location with well-known techniques of the prior artthat use laser detection of a touch over flat and light color surfacesin virtual keyboard projection applications. However, this uneventexture surface and the soft material movement during finger touches maybe used to improve the detection accuracy of the tactile gestures insome embodiments of the present invention as further detailed below inconjunction with FIGS. 8A-8C and 9.

FIG. 2 illustrates a simple tactile gesture by which a smartwatch 40 insome embodiments can be used for selecting an action marked by a groupof icons 92, 94, 96 and 98. The user touches at point 86 which is at theright lower quadrant of the virtual touchpad 80 that is generallycorresponding to ICON4 98 on the display. Thus, in this example, ICON4is selected and if the user taps at point 86, the action associated withICON4 is performed. This icon selection process is capable to workproperly even with some significant inaccuracy in the determination ofpoint 86.

FIG. 3 demonstrates a text input mode wherein the user draws eachcharacter 132 on the virtual touchpad 80 with the index finger 116. Asthe process continues, the display 42 shows the characters recognized sofar 130 as well as the currently drawn character 134. In the example ofFIG. 3, the user is entering the sentence “[you can st]art now”, wherethe bold characters within the brackets are already recognized by thesystem, and the current character entry is “a”. Standard contextprediction (e.g. auto-completion) techniques may speed up the entryprocess.

The text input operation of FIG. 3 requires complex tactile gestures,that include touch, drag, and hover operations performed by theoperator's fingers. Such complex tactile gestures include multiplesub-gestures that must be performed in a predefined order.

FIG. 4 shows how a common multi-finger gesture like a pinch along thedirection marked by the arrow 136 on the virtual touchpad 80 magnifiesthe display 42. The display 42 is proportionally magnified verticallyand horizontally as shown by arrow 138. The virtual touchpad in thisexample simulates the standard multi-touch gesture to create anappropriate tactile control command for the smartwatch, using twofingers 116 and 117. The blood vessels 108, 109, and 110 and skinblemishes 111 and 112 that were drafted on FIG. 1 are not shown forclarity in FIGS. 2, 3 and 4.

FIGS. 1-4 demonstrated various tactile gestures that may be used tocontrol the device 40. FIGS. 1, 2 and 4 showed very simple tactilegestures while FIG. 3 demonstrated quite a complex set of gestures thatare required for text entry recognition. Examples of simple tactilegestures may include tap, double-tap, long-tap, drag and hovering by oneor more fingers, as well as common multi-finger gestures like “pinch”,“rotation” and the like. We refer in this disclosure to the set of allpossible gestures that are recognizable by the device 40 as the set ofpredefined gestures. It should be understood that such a set can beextended either dynamically from one application to another, or byfrequent product updates, for example, via the internet.

FIG. 5 discloses an electronic device 40 like a smartwatch in accordanceto some embodiments of the present invention. The processor 10 providesthe processing and control means required by the system, and itcomprises one or more Central Processing Units (CPU). The CPU(s) insmall electronic devices are often referred to as the microprocessor ormicro-controller. The processor 10 uses the memory subsystem 15 forretaining the executable program, the data and the display information.In general, the memory subsystem 15 is a tangible and/or non-transitorycomputer-readable storage media for storing computer-executableinstructions or data structures. A motion sensor system 20 interfaceswith the processor 10 to provide ballistic data relating to themovements and rotations (tilt changes) of the device made by the user.The ballistic data can be used by the micro-controller to correlate withdetected finger touches on the virtual touchpad.

A display controller module 25 controls the display panel module 41 inaccordance with a program executed by the processor 10 and furtherinterfaces with the memory subsystem 15 for accessing the contents view.The display controller 25 may include local graphic memory resources.The display panel module 41 may be optionally equipped with touch screeninterface to receive the user's touch gestures. In such embodiments thedisplay controller 25 provides the processor 10 with the touch screengestures performed by one or more fingers on the display panel. When thedisplay stretches from one side of the device to the other, the touchsensing area may be extended beyond the boundaries of the display, sothat the device can respond to user touches on the side of the device'scase.

The motion sensor 20 detects rotational movements (or tilt gestures),movements and tap vibrations. There are many types of motion sensors,including gyroscopes, accelerometers, magnetic, mechanical, backgroundradio direction sensors, and more. The motion sensor system may comprisean assembly of one or more sensors of different types, with a special“sensor fusion” algorithm designed to improve the accuracy. Often, themotion sensor 20 includes built in computing elements to perform the‘fusion’ algorithm. For example, a 6-degree-of-freedom sensor, whichcomprises a combination of a 3-axis accelerometer and 3-axis gyroscopecan be used to distinguish between rotational and movement data andprovide more precise view navigation. It can also use accelerometer datato compensate for a gyroscope drift.

One or more cameras 50 may be employed to capture images of the virtualtouchpads and their immediate surroundings. The CPU(s) perform standardimage processing techniques on the captured images to extract thepositions of the operator's fingers relative to the virtual touchpads.This allows the CPU(s) to determine where the user touches or to detectfingers hovering on the virtual touchpads during complex tactilegestures like those disclosed in FIG. 3. The cameras may use regularlight or special waveform. As discussed below, the cameras may be placedon the side of the smartwatch viewing the back hand of the hand 100wearing the smartwatch 40.

In some embodiments, the smartwatch 40 may include a sound/finger tapsensor(s) 60 that may detect finger tap commands by the user, asdisclosed in my U.S. Pat. Nos. 6,466,198 and 6,933,923. When the usertaps or touches the virtual back hand touchpad 80, the sound andvibrations of the finger tapping are conducted by the hand itself untilit reaches the finger tap sensor(s) within the smartwatch enclosure. Aplurality of such sensors may be used to pinpoint the location of thefinger touch on the virtual touchpad 80.

The sound/finger tap sensor/s 60 may comprise a microphone and digitalsignal processing (DSP) interface to filter legitimate finger tap soundsand notify the processor 10. The DSP action may be performed by theprocessor 10, or may be performed by a built in microcontroller of a“smart” sound sensor. In other embodiments, the finger tap sensor mayuse a vibration detector like a piezoelectric finger tap sensor.

In some embodiments, the processor 10 performs a sensor “fusion” betweenthe finger tap sensor(s) 60 and the motion sensor 20 to validate thefinger tap commands. It correlates the sound (or vibrations) input fromthe finger tap sensor(s) 60 to the corresponding vibration captured bythe motion sensor 20. This arrangement can distinguish between a fingertap sound made by the hand wearing or holding the device and anotherfinger tap sound made by the other hand.

In some embodiments, the device 40 may include a radio system that actsas a radio touch sensor 70. The radio sensor sends an RF signal from atransmitting antenna via the user body to a set of receiving antennaethat localizes the touch position on the virtual back hand touchpad 80as described below.

The contents shown on the display may reside in the memory 15 or it canbe acquired from remote servers 34 in the cloud or via the internet 32.The connection to the internet or cloud is performed via an optionalnetwork interface 13 module of device 40. Actual data flows can be viaradio frequency or hardwired to a network plug. Alternatively, contentscan be acquired from external local media devices 30 that are connectedvia an optional input/output interface module 12 of the device.

The user may interface her smartphone 38 via the network interface 13,using Bluetooth or other interface techniques. Smartwatches may includebuttons 24 that are interfaced via the input/output interface module 12,to accept various user commands and input.

It should be apparent to a person skilled in the art that many variantsof the block elements comprising the block diagram of FIG. 5 can bemade, and that various components may be integrated together into one ormore VLSI chips. The processor 10 can optionally access additional userinterface resources such as a voice command interface. Various othercomponents are typically used in the device 40 that are not shownexplicitly in FIG. 5, but their existence should be recognized by aperson skilled in the art. These components may include power/batterysystems, RF communication circuitry, audio circuitry and other modules.

FIG. 6 illustrates how two regular or infrared cameras 50 and 52 areplaced on the side of the smartwatch 40 so that the cameras can detectfinger gestures on the virtual touchpad 80 on the back hand 100.Alternatively, a single camera 54 may be placed on the middle of theside of the watch. In some other embodiments, cameras may be placed onthe other side of the watch, e.g. 56, to allow the watch to be placed oneither hand, or even to have additional virtual touchpad on the leftside of the watch (when the user's arm is exposed, as illustrated belowin FIG. 15).

As discussed in conjunction with FIG. 1, the top surface of the hand isnot flat or smooth. It has visible blood vessels 108, 109 and 110, aswell as various skin artifacts and blemishes 111, 112 and 113. Thepresent invention makes use of these items to better determine the pointof touch of the user finger, as described below in conjunction withFIGS. 8A-C and 9.

The user may need to calibrate the spatial resolutions of the camera bytapping with one finger on two opposite corners of the virtual touchpad80 (or on all 4 corners), as described below in FIG. 11. Often thetouches on the virtual touchpad 80 do not need to be very accuratelyresolved by the camera sensors as control can be quite good even withsome significant inaccuracy in determining the exact touch location.Thus, some embodiments may use only one camera with a possible reducedaccuracy in the determination of the finger touch location. Many controlgestures can use the relative movement of the finger(s) touching thevirtual touchpad instead of depending on the exact point of touch.

When using the two cameras 50 and 52 as shown in FIG. 6, the systemreceives two simultaneous images that are processed to determine thepoint of touch in each image. The data extracted from the two images arethen combined to determine the finger touch locations. The two cameras50 and 52 may be placed as far away from each other as possible, toprovide the best stereoscopic analysis of the finger touch locations.

FIG. 7A illustrates the placement of the camera optical system 50 withinthe smartwatch 40 enclosure so that the camera monitors the back handvia the opening 140 on the side of the enclosure facing the back hand.An optical system 142 projects the view of the back hand onto the actualcamera sensor 144, so that the area of the virtual touchpad is captured.The optical system may include wide angle or fisheye lens systems andmirrors to achieve this goal or it may be implemented with softwareprocessing that allows the capture and analysis of the area of interest.To achieve low implementation costs, the cameras may be generallyselected with the lowest resolution that is still capable to locate thepoints of touch (e.g. 84, 86) of the user's finger(s).

FIG. 7B shows the effective viewing cone 150 that is created by theoptical system 142 in order to capture the finger touch locations. Asthe smartwatch 40 is worn on the wrist area 104, the viewing cone 150 isdirected down towards the back hand area 106. Various features visibleon the back hand like blood vessels 108 or other natural markers 110 maybe used to better relate the finger touch point relative to the backhand area.

The user may naturally raise the back hand area relative to the camera50 during operation. The system needs to constantly correct for suchchanges in the angle at which the camera looks at the back hand area.FIG. 7C shows how the back hand area may be tilted upwards to provide abetter view of the back hand surface area for the cameras.

In some embodiments with cameras, the slight movements of the skinsurface over the virtual touchpad area occurring when the user touchesthe back hand surface may be used to improve the touch detection processas illustrated in FIGS. 8A-8C. Each of these figures illustrate thecaptured image 115 that shows details of the assigned virtual touchpadsurface area 80, the finger 116, and the various blood vessels 108, 109and 110 and blemishes 111, 112, 113. Throughout this disclosure, werefer to the blood vessels, the blemishes and other texture variances assurface artifacts. While the captured image 115 is shown as having arectangle shape, it should be understood by any person familiar with theart of image processing that the shape can be different, e.g. ellipsoid,based on the specific implementation of the optical system 50.

FIG. 8A illustrates the captured image 115 when the finger 116 ishovering over the intended touch point. Notice that the surfaceartifacts 108,109, 110, 111, 112, and 113 appear to be undisturbed,because the finger is not yet depressing the skin. Therefore, the systemregards the situation depicted in FIG. 8A as “Hover”. FIG. 8Billustrates what happened when the finger 116 actually touches the skinof the back hand. Blood vessels 110 and 108 slightly move to positions117 and 119 in response to the touch. Similarly, blemish 112 slightlymoves to position 120. The processing of the captured image 115 detectsthese minute changes and updates the situation from “Hover” to “Touch”.

Assuming that the user is performing a drag gesture, FIG. 8Cdemonstrates visible changes in the surface artifacts of the back handand the position of the finger 116. Now that the finger is dragged inthe direction of the arrow, blood vessels 108 and 110 are restored totheir undisturbed positions, as the finger 116 is no longer affectingthem. Currently, during this drag operation, the captured image 115 ofFIG. 8C shows minute movements of the blood vessel 109 to 122, andblemish texture 120 to 121. The processing of the captured image 115continuously detects these minute changes on the surface artifacts, thusconfirming that the finger 116 is actively touching (and therefore,depressing) the back hand surface. The overall movement of the finger isrecorded and analyzed as a drag gesture. Thus, a simulated tactile dragcommand is provided to the smartwatch.

The process of finger touch detection 250 of FIGS. 8A-8C is performed insome embodiments along the flow diagram shown in FIG. 9. At step 252, animage of the fingers and the associated virtual surface is captured.This image is processed in step 254 to extract the position of one ormore finger tips over the virtual touchpad's surface. At the same time,step 256 extracts any positional and shape changes in the surfaceartifacts (blood vessels, blemishes, wrinkles and textures) adjacent tothe current finger tips positions. Decision step 260 determines if thesurface artifacts change in correlation to the movement of the fingerover the virtual touchpad. If changes in the surface artifacts aredetected and determined to be related to actual finger touch as explainin conjunction with FIG. 8A-8C, step 264 extracts a sequence of thefinger positions to detect a valid tactile gesture.

Step 268 of FIG. 9 continuously analyzes the sequence of the fingertouch positions. When decision step 270 confirms that a viable touchgesture was performed by the user, the gesture detection is concluded.Step 274 then simulates a tactile input command based on the detectedgesture. This input command is provided to the smartwatch and the devicescreen or its functions are changed accordingly. The process of steps252, 254, 256, 260, 264, 268, and 270 may be run continuously in a loop,depending on the operation mode of the smartwatch device.

In some embodiments, the smartwatch may employ a plurality of soundsensors as shown in FIG. 10A. Here there are four sound sensors 60, 62,64 and 66 which are placed close to the corners of the smartwatch nearthe bottom of the enclosure, as illustrated in FIG. 10B. Thus, thesensors are close to the wrist area 104. The four sound sensors mayemploy an array-based detection technique to determine the location ofthe touch on the virtual touchpad 80. The sound detectors may becombined with a motion sensor to verify and better analyze the taplocation as discussed with FIG. 12 below.

FIG. 11 shows a flow diagram of the calibration of the virtual touchpad80 placement on the back hand. Step 300 captures the finger positionusing cameras or sound sensors or a combination thereof. Decision step310 checks if a calibration of the virtual touchpad 80 has already beenmade. If not, the system prompts the user in step 314 to tap corners ofthe virtual touchpad on the back of the hand. The user may tap twoopposite corners or all four corners of the desired virtual touchpadarea. The system accepts the input in step 318, and creates a relativetransformation to the display by correlating the corners of the displayto the corner points entered by the user. Step 320 may optionallycapture, if cameras are used, the relative positions of the varioussurface artifacts (blood vessels 108, 109, and 110 and other naturalblemishes 111, 112, and 113) viewable on the back hand virtual touchpad.These artifacts may be used to assist in the touch point detection asillustrated above. The system also marks a calibration done flag in step324 so that the calibration process is not repeated. Step 330 convertsthe detected finger positions on the virtual touchpad into relativeposition information that maps the finger position on the virtualtouchpad onto the corresponding relative position on the device'sdisplay. In some embodiments with more than one virtual touchpads, someof the virtual touchpad may be calibrated to a non-related area shapethat is not directly mapped to the actual shape of the device display.This occurs when the particular virtual touchpad is used for number oftaps detection only, e.g. in the complementary mode discussed inconjunction with FIG. 16 below.

While the previous discussion indicated how to detect the finger touchlocations using different methods, it is possible that the smartwatchdevice will include several sensors for various functions. For example,as shown in my pending U.S. application Ser. No. 15/394,643, asmartwatch may use wrist tilting for view scrolling with finger tappingto input commands. Such a smartwatch will include a motion sensor 20 anda sound/finger tap sensor 60 to perform this operation. The smartwatchmay also employ the cameras 50 for the present invention. Theavailability of multiple sensors allows increase accuracy of detectingfinger touch location. FIG. 12 illustrates how data from various sensorsthat may be employed in some embodiments can be correlated to improvethe accuracy of determining the finger touch positions.

Multiple sensors may be grouped together to obtain an overall betteraccuracy than when using only one type of sensor. In some embodiments,the smartwatch embodiment relating to FIG. 10 includes cameras 50, soundsensors 60 and motion sensor 20. Step 350 captures finger positionsusing sound sensor as described in FIG. 8A. Step 354 monitors the motionsensor data and correlates the data with the sound data. Decision step360 looks at the correlation and if there is no good correlation, it mayuse the data from the sound sensors but mark the detection accuracy aslow in step 386. Alternatively, it may discard the data if thecorrelation between the motion data and the sound data is very bad. Step370 capture finger(s) position using the cameras, independently to step350. If step 360 indicated that there is a good correlation between thesound data and the motion data, the finger touch positions computed bystep 370 from the cameras are compared with the positions computed bythe sound and motion vibration data. If the data correlates well,decision step 380 marks the detection accuracy as high in step 384. Thedata is then used in step 390 to control the smartwatch device. If thereis no good correlation, decision step 380 marks the accuracy as low instep 386.

Differences between the sound data and the camera data may be used toimprove each other if common predictive estimation techniques are used.FIG. 12 (without steps 370 and 380) illustrates how sound sensors andmotions sensors can be used together to improve the accuracy of thedetection of the touch location when there are no cameras 50 in thedevice. Similarly, in some embodiments, the cameras may be combined onlywith a motion sensor 20 to detect vibration during a tap on the virtualback hand touchpad.

The arrangement in FIG. 12 may obtain occasional positioning fixesduring periods of time that the detection accuracy is marked high instep 384. In the other time periods, the camera vision detects thefinger position at a lower accuracy but assesses the correct relativeposition to previously accurately captured positions. Various controlestimation and correction techniques known in the art may be employed tofurther improve the accuracy of previously recorded sequence of fingerpositions based on these intermittent more accurate positioning data.For example, position sequence data may be recorded as “estimation” whenthe detection accuracy is marked as low by step 386. During intermittenttimes when higher accuracy is indicated by step 384, the entire“estimation” sequence that has been recorded so far is adjusted based onthe higher accuracy positions.

The cameras 50 and the sound/tap sensors 60 described above are passivesensors, where the system determines the touch location(s) on thevirtual back hand touchpad 80 from what it “sees” or “hears”. In someembodiments, the device 40 may include a radio system that act as aradio touch sensor 70, as shown in FIG. 13. The radio sensor sends an RFsignal from one or more transmitting antenna 72 via the user body to aset of receiving antennae 74 and 76 that localizes the touch position onthe virtual back hand touchpad 80. FIG. 13 illustrates the optimizedgain map 160 of the transmitter showing that the transmitter isgenerally directed towards the body of the user. Similarly, theoptimized gain maps 162 and 164 for the receivers 74 and 76,respectively, are directed towards the back hand area with the virtualtouchpad.

FIG. 14 illustrates how signals are being transmitted and receivedbetween the transmitter 72 and the receivers 74 and 76. The user's bodyperforms as a communication channel 200 connecting the two hands. Thetransmission signal along path 220 reaches the right hand 102 and thetouch point 88 over the virtual touchpad 80. From the touch point 88,the transmission reaches the receiver via paths 224 and 226. Even thoughthe antennae gain map directs the transmission and receiving as shown,some intensity of the transmitted signal from the transmitter 72 reachesthe receivers 74 and 76 directly along the short paths 230 and 232. Theradio sensor compares the arrival time from path 224 and 226 that arechanging in triangulation of the touch point 88, with the arrival timeof fixed delay via paths 230 and 232 to determine the touch point 88location over the virtual touchpad 80.

The transmission signal from transmitter 72 should be extremely low toconserve battery. It is designed to use frequencies that well propagateover the skin of the user's body 200 from the device 40 to the virtualtouchpad 80. To avoid the need of constant transmission, the radiosensor 70 may be kept normally inactive, until a user command activatesthe virtual touchpad 80. One method to do so is by having the user tapthe back hand, assuming the device 40 includes the sound/finger touchsensor.

Other active methods to determine the touch location may use ultrasoundvia the wrist carrying the watch and observing the echo signals with anarray of sensor. Alternatively, vibrations may be induced at variousfrequencies in the left hand so that detecting any changes induced bythe touch of the finger on the back hand can be used to localize thetouch position.

In some embodiments, it is desirable to have cameras on both sides ofthe smartwatch as shown in FIG. 15, where camera 54 is looking at theright side of the watch (as described in previous embodiments), andcamera 56 is looking at the left side of the watch. Camera 56 may bekept off unless the user wears the watch on the other hand so thatcamera 56 faces the virtual back hand touchpad. If the watch design isvertically symmetric, camera 54 would be sufficient when the user usesthe other hand as the watch may be rotated to keep camera 54 pointing tothe virtual back hand. If the watch design is not vertically symmetric,namely, the watch screen has distinguished top and bottom sides, camera56 is needed to monitor the virtual back hand touchpad.

In some embodiments, camera 56 is used for a second virtual touchpad 90placed on the arm side of the watch. When virtual touchpad 90 that isassigned over the arm is exposed, the user may provide command entryfrom both the left virtual touchpad and the right virtual touchpad. Ifthe arm is covered by a sleeve, the area above the sleeve may beassigned for the second virtual touchpad 90. The movement of theartifacts of texture on the sleeve surface may assist in detectinghovering and touching situation as described in conjunction with FIG. 8.In some embodiments, both virtual touchpads may be used interchangeably,so that each virtual touchpad may recognize all the gestures. In otherembodiments, a complementary mode may be employed, where each virtualtouchpad may recognize only a subset of the gestures. Thus, complexgestures, like the text entry operation of FIG. 3 that includes multiplesub-gestures, may be performed only on the back hand pad 80. Somegestures during complementary mode may need to be entered specificallyon the left or the right virtual touchpad at a predefined order. Forexample, some complementary mode gestures may include left tap and righttap combinations at some preset order. The operation modes when usingtwo or more virtual touchpads are discussed in more detail in thefollowing discussion of FIGS. 16-18.

FIG. 15 shows only a single camera on each side. It should be understoodthat two cameras (or more) may be placed on each side for stereoscopicdistance measurements to the touch locations as explained in FIG. 6.

FIG. 16 demonstrates how the teaching of the present invention can beutilized in the case of a general device 540 with a display 542 placedon an arbitrary surface 500, e.g. a low profile device placed on a tablesurface. The arbitrary surface may be soft or may have texture artifactsthat move or change when the user actually touches the surface. Thedevice 540 has multiple cameras placed along one or more sides of thedevice. FIG. 16 demonstrates a device 540 that uses cameras on all foursides in order to use four virtual touchpads 502, 504, 506 and 508. Twocameras 550 and 552 are placed at the bottom side of the device, and twocameras 558 and 560 are placed at the top side of the device. Forillustration purposes, the right and left sides use only one cameraeach, 554 and 556 respectively. The viewing cones of bottom side cameras550 and 552 are 570 and 572 respectively, and the viewing cone of theright side camera 554 is 574. For clarity, the viewing cones for theother cameras on the top and left sides are not shown in FIG. 16.

The virtual touchpads are assigned on the arbitrary surface 500 so thatthey generally correspond to the associated side of the device. However,the size and shape of each virtual touchpad can be different. They canalso change dimension dynamically during operation. In FIG. 16, this isdemonstrated in having the top virtual touchpad 508 made smaller andplaced skewed to the right relative to the top side and at a greaterdistance from the device compared to the bottom virtual touchpad 502. Tofurther demonstrate the ability to use unequal virtual touchpadassignments, the right virtual touchpad 504 is smaller than the leftvirtual touchpad 506.

The virtual touchpads may be configured for interchangeable operationmode or for complementary operation mode. In interchangeable operationmode, each virtual touchpad interfaces with the display 542 in a similarfashion to the operation detailed in FIGS. 1-4 above. Thus, the user maytap virtual touchpad 504 to perform some functions, and then performother functions tapping on virtual touchpad 502 or 506. In complementaryoperation mode, the virtual touchpads may be configured so that somevirtual touchpads perform only tap commands, while other touchpads allowmore complex drawing and pinching gestures. Some user interfaceprotocols may include entry on multiple virtual touchpads. Thus, forexample, one command can be entered if the user tap once on the lefttouchpad 506 followed by tapping twice on the right virtual touchpad504. Another command is simulated when the user taps in opposite order,namely first a tap on virtual touchpad 504 followed by two taps onvirtual touchpad 506.

FIG. 17 is a flow diagram of N virtual touchpads working in theinterchangeable operation mode 600. At step 605, all the N virtualtouchpads are assigned their surface areas and are calibratedindividually as described in FIG. 11. In Steps 610, 614 and 620, theactive touchpad is determined as the virtual touchpad on which the usertouches. At that point, the corresponding active touchpad is marked insteps 612, 616 and 622, and step 630 simulates the virtual touchpadactivity, based on the geometries of the active touchpad. Step 630 maylast a pre-defined period of time or continue to perform while useractivity is continuously detected. The activity ends at step 635 whenthe user function is fully performed or at the end of the pre-definedperiod of time when no further user activity is detected. Step 635 alsomarks all touchpads inactive and continues to monitor all the touchpadsfor new activity.

The flow diagram for the complementary operation mode 650 of N virtualtouchpads in some embodiments is illustrated in FIG. 18. Again, at step655, all the N virtual touchpads are assigned their surface areas andare calibrated individually as described in FIG. 11. In this mode allthe virtual touchpads may work at any time (that is, their cameras'input is processed at all time), as the user may tap different sequencesof touchpads to perform a complex complementary gesture. Therefore, step660, 662 and 668 monitor the different virtual touchpads for activity.Specific activities relating to the assignment of each virtual touchpadare performed in steps 670, 672, and 678. Each activity may depend oninput from a different virtual touchpad when the user creates acomplementary mode gesture comprising sequence of taps on differentvirtual touchpads. At step 680, the overall device activity isperformed, based on the multiple inputs from the various touchpads.

Some embodiments of the present invention may include tangible and/ornon-transitory computer-readable storage media for storingcomputer-executable instructions or data structures. Such non-transitorycomputer-readable storage media can be any available media that can beaccessed by a general purpose or special purpose computer, including theprocessors discussed above. By way of example, and not limitation, suchnon-transitory computer-readable media can include RAM, ROM, EEPROM, orany other medium which can be used to carry or store desired programcode means in the form of computer-executable instructions.Computer-readable medium may be also manifested by information that isbeing transferred or provided over a network or another communicationsconnection to a processor. All combinations of the above should also beincluded within the scope of the computer-readable media.

Computer-executable instructions include instructions and data whichcause a general purpose computer, special purpose computer, or specialpurpose processing device to perform a certain function or group offunctions. Computer-executable instructions also include program modulesthat are executed by computers in stand-alone or network environments.Generally, program modules include routines, programs, components, datastructures, objects, abstract data types, and the functions inherent inthe design of special-purpose processors, etc. that perform particulartasks. Computer-executable instructions, associated data structures, andprogram modules represent examples of the program code means forexecuting steps of the methods disclosed herein. The particular sequenceof such executable instructions or associated data structures representsexamples of corresponding acts for implementing the functions describedin such steps.

The embodiments described above may make reference to specific hardwareand software components. It should be appreciated by those skilled inthe art that particular operations described as being implemented inhardware might also be implemented in software or vice versa. It shouldalso be understood by those skilled in the art that differentcombinations of hardware and/or software components may also beimplemented within the scope of the present invention.

The description above contains many specifications, and for purpose ofillustration, has been described with references to specificembodiments. However, the foregoing embodiments are not intended to beexhaustive or to limit the invention to the precise forms disclosed.Therefore, these illustrative discussions should not be construed aslimiting the scope of the invention but as merely providing embodimentsthat better explain the principle of the invention and its practicalapplications, so that a person skilled in the art can best utilize theinvention with various modifications as required for a particular use.It is therefore intended that the following appended claims beinterpreted as including all such modifications, alterations,permutations, and equivalents as fall within the true spirit and scopeof the present invention.

1. A computer-implemented method for confirming a touch gesture betweentwo objects, the method comprising: capturing a sequence of images ofsaid objects; extracting from said captured images a sequence ofpositions of said objects; detecting a potential touch gesture betweensaid objects based on the proximity of their extracted positions;extracting from said captured images, when a potential touch gesture isdetected, a sequence of the shapes and the positions of one or moremovable visible surface artifacts in the vicinity of the position ofsaid detected potential touch gesture on at least one of said objects;and confirming said potential touch gesture when change of positions orshapes of said artifacts is detected in correlation with said potentialtouch gesture.
 2. The computer-implemented method of claim 1, whereinsaid touch gesture is a stationary touch gesture.
 3. Thecomputer-implemented method of claim 1, wherein said touch gesture is adragging touch gesture.
 4. The computer-implemented method of claim 1,wherein said potential touch gesture is determined to be a hoveringgesture when said extracted shapes or positions of said artifacts arenot changing in correlation with said potential touch gesture during thetime that said potential touch gesture is detected.
 5. Thecomputer-implemented method of claim 4, further comprising recognizingcomplex gestures of one object over the other object, said complexgestures comprising intermingled sequences of touch gestures andhovering gestures.
 6. The computer-implemented method of claim 5,wherein said complex gestures are text writing gestures of one objectover the other object.
 7. The computer-implemented method of claim 1,wherein one of said objects with movable surface artifacts is the skinsurface over an assigned body area of a user and wherein the otherobject is a finger of said user.
 8. The computer-implemented method ofclaim 7, wherein said one or more moveable surface artifacts areselected from the group consisting of blemishes, blood vessels,wrinkles, hair, stains, or any other visible textured section of saidskin surface.
 9. A system configured for detecting a touch gesturebetween two objects, comprising: one or more processors; one or morecameras coupled to said one or more processors and configured to captureimages of said objects; memory; and one or more programs, wherein theone or more programs are stored in the memory and configured to beexecuted by the one or more processors, the one or more programscomprising instructions for: extracting from said captured images asequence of positions of said objects; detecting a potential touchgesture between said objects based on the proximity of their extractedpositions; extracting from said captured images, when a potential touchgesture is detected, a sequence of the shapes and the positions of oneor more movable visible surface artifacts in the vicinity of theposition of said detected potential touch gesture on at least one ofsaid objects; and confirming said potential touch gesture when change ofpositions or shapes of said artifacts is detected in correlation withsaid potential touch gesture.
 10. The system of claim 9, wherein saidpotential touch gesture is determined to be a hovering gesture when saidextracted shapes or positions of said artifacts are not changing incorrelation with said potential touch gesture during the time that saidpotential touch gesture is detected.
 11. The system of claim 10, whereinsaid one or more programs further comprising instructions forrecognizing complex gestures of one object over the other object, saidcomplex gestures comprising intermingled sequences of touch gestures andhovering gestures.
 12. The system of claim 9, wherein one of saidobjects is the skin surface over an assigned body area of a user, andwherein said other object is a finger of said user.
 13. The system ofclaim 12, wherein said system is a smart watch.
 14. The system of claim13, wherein said one or more cameras are placed on the smart phone'sside facing said skin surface, and wherein each camera furthercomprising an optical system enhancing side viewing capabilities toenable image capturing of said finger touching said skin surface. 15.The system of claim 9, further comprising one or more sound sensorscoupled to said one or more processors, and wherein said one or moreprograms further comprising instructions for: discovering a touchgesture using said one or more sound sensors; and confirming saiddiscovered touch gesture based on the correlation with said potentialtouch gesture detected by said cameras.
 16. The system of claim 9,further comprising one or more motion sensors attached to one or more ofsaid objects and coupled to said one or more processors, and whereinsaid one or more programs further comprising instructions for:discovering a touch gesture using said one or more motion sensors; andconfirming said discovered touch gesture based on the correlation withsaid potential touch gesture detected by said cameras.
 17. One or morenon-transitory computer-readable media encoded with computer programinstructions that when executed by one or more processors, cause the oneor more processors to confirm a touch gesture among two or more objectsby performing operations comprising: capturing a sequence of images ofsaid objects; extracting from said captured images a sequence ofpositions of said objects; detecting a potential touch gesture between apair of said objects based on the proximity of their extractedpositions; extracting from said captured images, when a potential touchgesture is detected, a sequence of the shapes and the positions of oneor more movable visible surface artifacts in the vicinity of theposition of said detected potential touch gesture on at least one ofsaid objects; and confirming said potential touch gesture when change ofpositions or shapes of said artifacts on at least one of said pair ofobjects is detected in correlation with said potential touch gesture.18. The one or more non-transitory computer-readable media of claim 17,wherein one of said objects is the skin surface over an assigned bodyarea of a user, and wherein said one or more other objects are thefingers of said user.
 19. The one or more non-transitorycomputer-readable media of claim 18, wherein said touch gesture is amulti-finger touch gesture over said skin surface.
 20. The one or morenon-transitory computer-readable media of claim 18, wherein saidpotential touch gesture is determined to be a hovering gesture when saidextracted shapes or positions of said artifacts are not changing incorrelation with said potential touch gesture during the time that saidpotential touch gesture is detected.
 21. The one or more non-transitorycomputer-readable media of claim 20, further comprising recognizingcomplex gestures of one object over the other object, said complexgestures comprising intermingled sequences of touch gestures andhovering gestures.